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Abstract 

Since  the  seminal  work  investigating  the  relationship  between  typieal  and 
maximum  performanee  by  Saekett,  Zedeek,  and  Fogli  in  1988,  there  has  been  a  marked 
inerease  in  researeh  in  this  area.  Although  mueh  researeh  has  furthered  the  relationship 
between  typieal  and  maximum  performanee,  none  have  attempted  to  identify  whieh 
leadership  effeetiveness  eriteria  are  eonsidered  most  important  to  an  individual’s 
maximum  performanee,  or  assessment  of  one’s  potential.  Thus,  this  empirieal  study 
seeks  to  identify  the  leadership  effectiveness  eriteria  under  maximum  performanee 
eonditions  as  it  relates  to  entry  and  middle  level  managers. 

Using  an  exploratory  factor  analysis,  the  results  suggest  an  interesting  eomparison 
of  leadership  eriteria  between  entry  and  middle  management  engaged  in  maximum 
performanee.  For  entry  level  managers,  personality,  effort,  and  attitude  emerged  as  the 
most  important  faetors.  Taken  together,  these  faetors  suggest  that  “leadership  of  self’ 
deseribe  the  pathway  to  being  an  effeetive  leader.  However,  for  middle  level  managers, 
trust,  aeeommodation,  and  adaptability  were  eonsidered  essential  leadership  effeetiveness 
eriteria  indicating  “leadership  of  team”  is  an  appropriate  framework  at  this  level. 
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LEADERSHIP  CRITERIA  UNDER  MAXIMUM  PERFORMANCE 

CONDITIONS 

I.  Introduction 

Over  the  past  eentury,  there  has  been  an  explosion  of  interest  in  the  topic  of 
leadership.  Often,  it  is  linked  to  performance  and  managerial  potential  (Ployhart,  Lim, 
&  Chan,  2001).  Moreover,  it  is  highly  regarded  as  critical  to  success  in  military 
environments  (Bass,  1998;  Campbell,  McHenry,  &  Wise,  1990).  Accordingly,  there  has 
been  a  great  deal  of  attention  in  both  academic  and  popular  literature  originating  from 
earlier  conceptions  captured  by  trait  theory  to  the  most  recent  and  highly  studied  model 
described  as  transformational  leadership.  Regardless  of  industry,  organizations  are 
fascinated  by  the  concept  of  leadership  and  its  ability  to  influence  others  to  achieve 
organizational  goals. 

Many  organizations  invest  resources  in  leadership  training  by  inviting  speakers  to 
share  their  ideas  on  leadership,  sending  their  employees  to  leadership  training  schools 
(i.e.  assessment  centers),  or  providing  other  activities  perceived  necessary  to  hone 
employee  leadership  skills.  Moreover,  managerial  performance  assessment  is  frequently 
based  on  the  ability  to  perform  job  functions,  work  effectively  in  teams,  and  lead  others. 
When  performance  is  measured  over  a  longer  period  of  time,  as  in  annual  performance 
appraisals,  the  type  of  performance  is  described  as  typical  performance  (Sackett,  Zedeck, 
&  Fogli,  1988).  However,  if  an  individual  is  being  evaluated  whereby  the  individual  is 
made  aware  of  the  evaluation,  the  highest  level  of  effort  is  expected,  and  the  assessment 
is  over  a  shorter  period  of  time  compared  to  typical  performance,  then  the  level  of 
performance  is  described  as  maximum  performance  (Sackett  et  ah,  1988). 
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The  underlying  goal  of  typieal  performance  is  to  determine  what  an  employee  will 
do  while  the  goal  of  maximum  performance  measures  is  to  determine  what  an  employee 
can  do.  Cronbach  (1960)  was  the  first  to  make  this  distinction  when  he  referred  to 
maximum  performance  as  “tests  of  ability”  and  typical  performance  as  what  an  individual 
is  “likely  to  do  in  a  given  situation  or  in  a  broad  class  of  situations”  (Cronbach,  1960, 
p.29).  This  distinction  is  important  especially  to  organizations  that  utilize  both  typical 
and  maximum  performance  methods  as  a  means  to  develop  and  select  employees  for 
hiring  or  advancement  potential. 

Understanding  which  dimensions  are  important  to  leadership  development  is  a 
key  component  to  understanding  and  improving  organizational  effectiveness  (Benson, 
2007).  However,  previous  research  does  not  seem  to  address  which  leadership 
dimensions  are  considered  fundamentally  relevant  to  maximum  performance  conditions. 
Instead,  much  of  the  attention  has  only  been  devoted  to  deductively  testing  the  impact  of 
a  priori  dimensions  under  these  conditions.  Moreover,  these  studies  are  often  analyzed 
by  asking  subjects  themselves  about  their  thoughts  and  perceptions  rather  than 
determining  which  leadership  dimensions  should  be  considered  relevant  to  maximum 
performance  in  the  first  place.  Accordingly,  little  work  seems  to  address  the  differences 
between  entry  and  middle  level  managers  exposed  to  maximum  performance  conditions 
which  is  only  possible  when  raters  are  asked  to  uncover  these  characteristics  themselves. 
Thus,  the  following  problem  statements  exist: 

What  are  the  leadership  effectiveness  criteria  under  maximum  performance 
conditions? 
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What  are  the  differences  in  leadership  effectiveness  criteria  between  entry  and 

middle  level  managers  under  maximum  performance  conditions? 

Methodology 

In  order  to  identify  potential  leadership  dimensions  that  pertain  to  maximum 
performance  conditions,  an  appropriate  venue  that  sufficiently  represents  the  conditions 
outlined  by  Sackett  et  al.  (1988)  must  be  selected.  Assessment  centers  have  frequently 
been  cited  as  fertile  ground  for  research  since  their  inception  during  the  1940s  (Lance, 
2008)  and  have  been  identified  as  appropriate  methodologies  to  simulate  maximum 
performance  conditions  (Ployhart  et  ah,  2001;  Bradley,  Nicol,  Charbonneau  &  Meyer, 
2002;  Lim  &  Ployhart,  2004).  Often,  they  are  used  for  personnel  selection  and  promotion 
(i.e.  administrative  assessment  centers)  and  training  (i.e.  developmental  assessment 
centers)  in  both  private  and  public  sectors  (Thornton  &  Rupp,  2006;  Lim  et  ah,  2004). 
Accordingly,  the  military  has  used  such  assessment  centers  as  a  mechanism  to  train  new 
and  experienced  officers  alike  to  develop  their  leadership  skills  and  select  top  performers 
as  a  means  of  recognition  and  career  benefit.  Such  an  application  of  assessment  centers 
has  been  acknowledged  by  the  research  community  and  simultaneously  serves  as  a  useful 
mechanism  to  study  leadership  under  maximum  performance  conditions.  Thus,  the  Air 
Force’s  leadership  training  school  for  entry  level  officers — Air  and  Space  Basic  Course 
(ASBC) — and  its  middle  manager  leadership  school — Squadron  Officer  School  (SOS) — 
were  the  developmental  assessment  centers  used  in  this  study  to  identify  leadership 
effectiveness  criteria  under  maximum  performance  conditions. 


3 


Often,  leadership  is  considered  an  important  component  to  an  individual’s 
organizational  performance  (Borman  &  Brush,  1993;  Campbell  et  ah,  1993).  Thus,  it  is 
essential  to  develop  appropriate  leadership  dimensions  that  can  be  used  to  accurately 
assess  the  criteria  considered  important  to  performance.  However,  the  process  of 
identifying  relevant  leadership  criteria  can  be  problematic  if  it  lacks  the  necessary 
academic  rigor  to  support  the  findings.  Accordingly,  a  deficient  process  could  easily 
create  espoused  constructs  that  misrepresent  such  leadership  effectiveness  criteria  and 
jeopardize  the  ability  to  assess  the  intended  leadership  dimensions  accurately  or  in  a 
meaningful  way.  Traditionally,  this  has  been  referred  to  as  the  construct  validity  problem 
(Lievens,  2002). 

There  has  been  much  debate  about  the  construct  validity  problem  associated  with 
assessment  centers.  Lance  (2008)  argues  that  high  correlations  commonly  observed 
between  constructs  and  exercises  suggest  that  assessment  centers  are  actually  measuring 
exercise-specific  behaviors  rather  than  construct-specific  behaviors.  Yet,  Howard  (2008) 
discusses  how  it  should  not  be  surprising  that  both  exercises  and  constructs  have  been 
observed  to  covary  because  exercises  are  designed  to  elicit  behaviors  that  can  be  used  to 
rate  intended  constructs.  Moreover,  Howard  (2008)  suggests  that  since  exercises  can 
elicit  both  intended  as  well  as  unintended  behaviors,  it  is  quite  possible  for  ratings  to 
reflect  both  intended  and  espoused  constructs. 

Whether  constructs  have  been  observed  to  reflect  exercises  or  not,  Arthur,  Day,  & 
Woehr  (2008)  suggest  that  raters  and  assessment  center  research  seem  to  accept  the  use 
of  espoused  constructs.  Furthermore,  they  argue  the  root  of  the  construct  validity 
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problem  is  the  inappropriately  developed  and  tested  eonstruct  development  proeess  for 
assessment  centers.  As  a  means  to  alleviate  this  concern,  Arthur  et  al.  (2008)  propose  the 
idea  that  assessment  centers  need  to  ensure  that  each  construct  intended  to  be  measured  is 
subject  to  the  rigor  of  a  psychometric  test  to  ensure  the  constructs  intended  to  be  rated  are 
in  fact,  being  rated.  Accordingly,  this  study  followed  an  academically  rigorous  procedure 
(Thornton  &  Rupp,  2006)  in  sync  with  the  most  contemporary  research  of  dimension 
development  to  derive  appropriate  leadership  criteria  and  thus,  avoid  the  rating  of 
espoused  constructs. 

Implications 

Identifying  leadership  criteria  under  maximum  performance  conditions  for  entry 
and  middle  managers  is  significant  to  both  practitioners  and  academics  alike.  There  has 
not  been  a  study  to  date  that  has  gained  insight  on  leadership  criteria  relevant  to 
maximum  performance  conditions.  While  there  have  been  studies  that  analyzed  a  priori 
leadership  dimensions  under  such  conditions,  there  have  yet  to  be  any  that  provide 
answers  to  the  question  of  which  leadership  dimensions  are  considered  important  in  the 
first  place.  Needless  to  say,  there  has  also  not  been  such  a  study  that  has  leveraged 
trained  experts  to  uncover  maximum  performance  leadership  dimensions — a  critical 
factor  to  ensuring  only  appropriate  dimensions  are  developed.  Moreover,  there  has  not 
been  a  study  that  has  distinguished  leadership  effectiveness  criteria  under  maximum 
performance  conditions  between  entry  and  middle  levels  of  management.  Such  a 
contribution  to  the  research  may  provide  invaluable  insight  for  practitioners  to  not  only 
differentiate  between  entry  and  middle  level  managers’  leadership,  but  also  provides 
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information  on  what  entry  and  middle  level  managers  ean  do  at  a  speeified  level  of 
experienee.  Aeeordingly,  the  results  of  this  study  can  be  used  to  help  shape  the  selection, 
promotion,  and  placement  process  of  entry  and  middle  level  management  within 
organizations.  Finally,  this  study  sheds  light  on  maximum  performance  research  by 
introducing  significant  leadership  analysis  to  the  body  of  knowledge. 
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II.  Literature  Review 


Originally  conceived  by  Cronbaeh  (1960)  and  popularized  by  Saekett  et  al. 
(1988),  researchers  have  eontinuously  tried  to  gain  insight  into  the  relationship  between 
typieal  and  maximum  performanee.  While  maximum  performanee  eonditions  were 
initially  studied  using  objeetive  measures  of  speed  and  aecuraey  of  eashiers’  ability  to 
eheekout  customers  during  peak  supermarket  eonditions  and  under  supervisor  review 
(Saekett  et  al.  1988),  there  has  been  a  wide  variety  of  researeh  to  attempt  to  further  the 
discussion. 

Maximum  vs.  Typical  Performance 

As  briefly  diseussed  earlier,  maximum  performanee  refers  to  what  an  individual 
ean  do  over  a  shorter,  speeified  period  of  time  and  typical  performance  refers  to  what  an 
individual  will  do  in  the  long  run.  Saekett  et  al.  (1988)  proposed  the  often  eited  three 
eonditions  considered  neeessary  to  exert  maximum  performanee:  1)  the  individual  is 
made  aware  of  an  evaluation  2)  the  individual  must  understand  that  maximum 
performanee  is  expected  and  therefore  exhibit  maximum  effort,  and  3)  there  must  be  a 
short  enough  time  span  sueh  that  the  individual  is  eapable  of  providing  maximum  effort. 
Regarding  typical  performance,  Saekett  et  al.  (1988)  deseribe  such  conditions  as 
performance  evaluated  over  a  regularly  seheduled  period  of  time  (e.g.  annual 
performanee  report)  whereby  maximum  effort  is  a  eonseious  endeavor,  strictly 
communicated  to  the  individual,  and  monitored  throughout  the  rating  period.  Saekett  et 
al.  (1988)  supported  their  findings  for  the  relationship  between  maximum  and  typieal 
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performance  using  the  results  from  a  supermarket  experiment  in  which  cashiers  were 
exposed  to  objective  evaluations  of  speed  and  accuracy  of  checkout  via  a  short  term 
sampling  of  the  job  and  a  30  day  review.  Their  findings  suggested  very  low  correlations 
in  performance  between  short  term  (i.e.  maximum  performance)  and  long  term  (i.e. 
typical  performance)  evaluations  (.14  and  .32  for  the  speed  evaluation  and  .11  and  .17  for 
accuracy). 

In  a  more  recent  commentary  on  the  seminal  Sackett  et  al.  (1988)  article,  Sackett 
(2007)  seeks  to  provide  additional  clarification  of  the  three  conditions  necessary  for 
maximum  performance.  In  hind  sight,  he  confesses  that  instead  of  using  the  word 
“necessary”  originally  used  in  1988  to  describe  the  three  conditions,  he  prefers  the  word 
“sufficient”  be  used.  Moreover,  he  mentions  the  intentionally  vague  “short  duration” 
condition  to  avoid  placing  overly  restrictive  constraints  and  to  provide  opportunities  to 
research  this  topic.  Thus,  the  three  conditions  help  grant  assurance  that  the  performance 
in  question  is  indeed  most  likely  to  be  a  maximum  performance  and  that  if  any  conditions 
are  absent,  it’s  not  that  the  potential  for  maximum  performance  is  lost,  but  that  that  it  is 
reduced  to  an  unknown  state  (Sackett,  2007).  However,  because  this  study  leveraged  the 
earlier,  more  restrictive  definition  of  maximum  performance,  there  is  little  doubt  in  the 
ability  of  ASBC  or  SOS  to  represent  maximum  performance  conditions. 

The  discussion  of  maximum  and  typical  performance  has  focused  on  a  wide 
variety  of  applications  as  highlighted  by  Dubois,  Sackett,  Zedeck,  and  Fogli  (1993)  to 
include  the  implications  of  generalizing  typical  performance  to  maximum  performance 
situations,  problems  associated  with  maximum  and  typical  performance  (Campbell  et  ah. 
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1990;  Guion,  1991;  Sackett  &  Larson,  1990),  utility  analysis  (Boudreau,  1991), 
implications  of  job  knowledge  and  cognitive  ability  tests  (Ackerman  &  Humphreys, 

1990;  Dubois  et  ak,  1993),  confounds  affecting  criterion  validity  (Borman,  1991),  the 
five  factor  model  of  personality  and  transformational  leadership  (Ployhart  et  ah,  2001; 
Lim  et  ah,  2004),  personality  and  cognitive  ability  (Bradley  et  ah,  2002;  Marcus,  Goffin, 
Johnston,  &  Rothstein,  2007),  maximum  performance  dimensions  as  an  accurate 
representation  of  task  priorities,  situational  constraints,  and  criterion  data  sources 
(Mangos  &  Arnold,  2008),  impact  of  “dark”  and  “light”  constructs  (Benson  &  Campbell, 
2007),  and  a  host  of  other  a  priori  psychological  constructs  investigated  across  the 
maximum  and  typical  performance  continuum  (Scholtz  &  Schuler,  1993).  While  many 
studies  have  attempted  to  expand  on  maximum  and  typical  performance,  no  studies  to 
date  have  included  an  inductive  study  to  determine  leadership  criteria  under  maximum 
performance  conditions  or  as  these  criteria  relate  to  entry  and  middle  management 
leadership. 

Assessment  Centers 

As  previously  mentioned,  assessment  centers  have  been  shown  to  be  an 
appropriate  venue  to  represent  maximum  performance  conditions.  Accordingly,  ASBC 
and  SOS  simulate  such  a  performance  environment  as  determined  by  Sackett  et  al.  (1988) 
for  both  entry  and  middle  level  managers.  ASBC  is  a  six  week  training  program  that 
expressly  encourages  entry  level  managers  to  maximize  their  performance  throughout  the 
course.  SOS  is  a  five  week  course  that  parallels  the  characteristics  of  ASBC  but  is 
tailored  to  middle  level  managers.  Moreover,  a  distinguished  graduate  program  exists  in 
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each  assessment  center  to  reward  the  highest  performing  individuals.  Beyond 
encouragement  from  assessment  center  staff,  the  distinguished  graduate  program  further 
incentivizes  students  to  perform  to  the  best  of  their  ability  because  it  nearly  guarantees 
advancement  for  the  next  10  years.  Thus,  the  students  at  each  assessment  center  are 
made  acutely  aware  of  their  evaluation,  explicitly  encouraged  to  exert  their  highest  effort, 
and  are  assessed  over  a  considerably  shorter  duration  than  annual  performance  reports 
that  capture  typical  performance.  Taken  together,  clearly  the  assessment  center 
conditions  at  ASBC  and  SOS  fit  the  maximum  performance  conditions  outlined  by 
Sackett  et  al.  (1988). 

Leadership  Dimension  and  Criteria  Development 

While  studies  have  indicated  the  usefulness  of  assessment  centers  (Gaugler, 
Rosenthal,  Thornton,  &  Benson,  1987;  Arthur,  Day,  McNelly,  &  Edens,  2003),  one  of  the 
important  considerations  that  should  precede  their  use  is  the  issue  of  construct 
development,  or  dimension  development  as  it  is  often  referred  to  in  an  assessment  center 
setting  (Thornton  &  Rupp,  2006).  As  previously  mentioned,  without  a  meticulous 
methodology  to  develop  appropriate  dimensions  for  assessment  centers,  there  is  a 
significant  risk  of  rating  dimensions  that  inaccurately  represent  leadership  effectiveness 
criteria  (Arthur  et  ah,  2008).  By  engaging  in  a  rigorous  dimension  development  process 
for  ASBC  and  SOS,  this  study  intends  to  alleviate  this  concern  and  mitigate  the 
possibility  of  rating  espoused  dimensions  instead  of  the  actual  dimensions  intended  to  be 
rated  (Arthur  et  ah,  2008). 
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A  variety  of  procedures  have  been  used  to  develop  appropriate  dimensions  to 
include  the  use  of  previous  research,  job  analyses,  surveys,  interviews,  and  questionnaires 
(Henderson,  1976).  For  example,  in  an  attempt  to  discover  which  dimensions  are 
important  to  assessment  centers,  Arthur  et  al.  (2003)  analyzed  179  articles  and 
manuscripts  which  led  to  the  identification  of  168  dimensions.  Additionally,  Arthur  et  al. 
(2008)  suggest  that  dimensions  should  be  developed  according  to  their  ability  to  reflect  a 
variety  of  validity  evidence  to  include  content,  criterion,  and  construct  validity  as  a  few 
types  of  evidence  to  consider.  Moreover,  it  has  been  suggested  that  quality  of  such 
validation  evidence  (e.g.  test-criterion  relationships,  content,  internal  structure  of  the  test, 
response  processes,  and  consequences  of  testing)  should  play  a  key  role  in  weighing  the 
contribution  of  each  source  to  validity  (SIOP,  2003).  Thornton  and  Rupp  (2006) 
highlight  that  an  important  first  step  to  identifying  dimensions  should  be  a  job  analysis. 

In  their  discussion,  they  discuss  how  an  assessment  center  job  analysis  is  broader  than  a 
traditional  job  analysis  in  that  an  assessment  center  job  analysis  extends  beyond 
knowledge,  skills,  and  abilities  (KSAs)  and  other  organizational  characteristics  (KSAOs) 
by  involving  “several  components  of  the  job  situation,  such  as  the  dimensions  required, 
the  relative  competency  required  for  each  dimension,  the  job  tasks,  and  the  organizational 
environment”  (Thornton  &  Rupp.,  2006,  p.81).  Thus,  they  argue  that  this  broader  sense 
of  job  analysis  should  be  used  because  of  its  ability  to  uncover  the  complexities  inherent 
in  an  assessment  program  (Thornton  &  Rupp,  2006).  When  considering  potential 
processes  to  use  for  dimension  development,  it  was  determined  that  a  compilation  of 
these  methods  would  be  the  most  appropriate  process. 
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Such  a  rigorous  process  is  consistent  with  the  one  proposed  by  Thornton  and 
Rupp  (2006)  in  which  potential  behavioral  dimensions  are  identified  by  eonducting  a  job 
analysis  using  information  about  the  job,  consulting  subject  matter  experts,  aequiring  and 
analyzing  data  stemming  from  expert  consensus,  and  doeumenting  the  findings  that 
emerge  from  the  common  factors.  In  particular,  they  point  out  that  suceessful 
assessment  centers  have  been  eareful  to  create  dimensions  that  were  rooted  in  a  job 
analysis  or  job  competeneies  composed  by  key  eontributors  to  performanee  (Thornton  & 
Rupp,  2006). 

The  strategy  proposed  by  Thornton  and  Rupp  (2006)  for  a  job  analysis  to  uncover 
KS As  and  KS AOs  can  also  be  applied  to  the  development  of  leadership  effeetiveness 
eriteria  insomuch  as  these  job  eriteria  are  the  relevant  KSAs  and  KSAOs  of  leaders. 
Regarding  key  eontributors  to  performance,  Thornton  and  Rupp  (2006)  consider  a 
multitude  of  methods  that  ean  be  used  ranging  from  direct  observation  or  participation  to 
interviews  with  subject  matter  experts,  trainers,  instructors,  or  educators.  They  consider 
experts  to  be  individuals  that  have  experience,  are  well  trained  and  eapable  of  providing 
insights  into  the  jobs  of  interest,  and  able  to  aecurately  rate  trainee  behaviors.  Moreover, 
there  should  be  a  means  to  provide  feedback  to  trainees.  Ideally,  these  experts  should 
reaeh  consensus  through  either  discussion  or  statistical  formula.  Furthermore,  they  assert 
that  preeautions  should  be  taken  to  mitigate  potential  biases  that  may  interfere  with 
collected  data  used  to  compose  appropriate  dimensions.  Onee  data  is  aequired,  it  must  be 
analyzed  to  formulate  appropriate  behavioral  dimensions.  Finally,  documentation  of 
dimension  development  is  an  important  step  to  create  flexibility  and  facilitate  the  periodic 
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review  of  assessment  center  design  and  the  associating  dimensions  deemed  relevant  to 
the  organization  (Thornton  &  Rupp,  2006). 

Experts  vs.  Novices 

When  it  comes  to  level  of  rater  expertise,  someone  with  a  high  degree  of 

proficiency  is  regarded  as  an  “expert.”  Contrarily,  one  who  lacks  proficiency  is  regarded 

as  a  “novice.”  Accordingly,  expertise  can  be  conceptualized  as  a  continuum  such  that 

experts  and  novices  are  at  each  extreme  (Chi,  2006)  and  proficiency  as  the  level  of 

training  and  experience  acquired.  Moreover,  Schenk,  Vitalari,  and  Davis  (1998,  p.l3) 

characterize  experience  in  terms  of  “episodic  knowledge”  which  is  defined  as: 

“...the  organized  collection  of  specific  job-relevant  events  or  situations  (i.e., 
episodes)  that  becomes  a  source  for  future  problem  solutions.  Moreover,  this 
episodic  knowledge,  if  it  exists  at  all  in  the  novice,  is  superficially  organized 
compared  with  that  of  the  experienced  professional. .  .Thus,  while  the  novice  may 
have  a  great  store  of  semantic  or  factual  knowledge,  including  the  latest 
techniques,  research  results,  and  critical  issues,  he  or  she  is  sorely  lacking  in 
grounded,  domain-specific  knowledge.” 

Schenk  et  al.  (1998)  also  suggest  that  experience  alone  is  insufficient  and  that  other 
factors  such  as  awareness  of  one’s  decision-making  and  ability  to  monitor  the  decision¬ 
making  process  are  also  important.  However,  training  can  be  considered  an  effective 
solution  to  refine  the  decision  making  process  and  bolster  decision  making.  Taken 
together,  training  and  experience  increase  proficiency  which  is  required  to  become  an 
expert. 
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Instructors  as  Experts 


When  deciding  which  audience  would  be  most  appropriate  to  collect  data,  it  was 
evident  that  ASBC  and  SOS  instructors  could  be  regarded  as  experts  given  their  level  of 
training  and  experience.  Upon  selection,  instructors  are  required  to  attend  five  weeks  of 
training  consisting  of  two  courses  and  a  week  of  supervised  instruction.  The  two  training 
courses,  theory  and  principles  of  adult  education  (TPAE)  and  practical  application  of 
adult  education  (PAAE),  are  each  two  weeks  long  and  cover  a  range  of  topics  including 
instructional  methodologies,  lesson  presentation  skills,  classroom  management, 
educational  evaluation,  assessment  processes  and  requirements,  and  supervised 
instruction  exercises.  The  fifth  week  of  training  requires  each  instructor  to  conduct  a 
class  under  the  supervision  of  a  fully  qualified  instructor  who  has  been  appointed  based 
on  ability  to  exhibit  the  highest  teaching  standards.  Once  the  five  weeks  of  initial 
training  are  completed,  additional  refresher  training  is  completed  during  an  instructor’s 
first  class,  annually  thereafter,  or  upon  request  of  instructors.  Eurthermore,  an  80  hour 
supplementary  training  course  and  other  workshops  are  provided  for  instructors  to 
improve  instructional  design,  curriculum  development,  presentation  and  delivery,  and 
assessments.  Throughout  the  extensive  training  provided,  instructors  are  expected  to 
demonstrate  subject  matter  expertise,  be  able  to  provide  quality  feedback  and  assessments 
of  students,  and  communicate  effectively.  To  enforce  these  standards,  instructors  are 
subject  to  mandatory  faculty  evaluations  by  supervisors,  their  students,  and  optionally  by 
peers.  These  evaluations  are  a  critical  feedback  mechanism  that  continuously  reinforce 
and  uphold  the  decision  making  framework  and  skills  demonstrated  by  experts. 
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Regarding  experience,  only  officers  with  at  least  four  years  of  time  in  service  for 
ASBC  and  eight  years  for  SOS  are  considered  for  instructor  selection.  Additionally, 
instructor  selection  is  contingent  on  the  ability  of  the  officer  to  meet  initial  qualifications 
consisting  of  previous  instructor  experience,  ability  to  exceed  performance  standards, 
professional  military  education  completion,  and  progress  or  completion  of  a  master’s 
degree.  Collectively,  the  initial  and  ongoing  training  and  experience  provide  instructors 
with  the  necessary  skills  to  continuously  refine  their  decision  making  mental  model  and 
exhibit  the  level  of  proficiency  that  is  indicative  of  an  expert. 

Advantages  of  Experts 

There  are  several  advantages  to  using  experts  in  this  study  as  a  means  of 
identifying  leadership  criteria.  Experts  have  been  observed  to  perform  better  under  time 
constraints,  more  accurately  than  non-experts,  and  are  able  to  structure  problems  and 
categorize  concepts  better  than  novices  (Chi,  2006;  Guerrero,  Gou,  and  Amau,  1997). 
Moreover,  Anderson  and  Lienhardt  (2002)  found  experts  could  identify  solutions 
immediately  and  were  able  to  recall  cognitive  rules  better  than  novices.  Gitomer  (1988) 
and  Glaser  (1990)  discuss  in  detail  the  advantageous  ability  of  experts  to  use  more 
accurate  mental  models  to  process  high  volumes  of  information  and  that  the  ability  to 
recall  information  is  at  a  faster  rate  than  less  skilled  individuals.  Keeney  and  von 
Wintderfeldt  (1991)  assert  that  “throughout  any  analysis,  expert  judgment  is  essential 
(p.l91).”  Thus,  given  the  choice  between  experts  or  novices  to  collect  data  on  leadership 
performance,  the  literature  clearly  favored  the  use  of  experts  which  in  turn,  more 
appropriately  described  the  instructors  consulted  in  this  study. 
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Disadvantages  of  Experts 


While  there  is  extensive  researeh  covering  the  advantages  of  experts,  there  is 
comparably  less  mention  of  the  limitations  of  experts.  Chi  (2006)  points  out  limitations 
such  as  how  experts  have  a  hard  time  adapting  to  areas  outside  their  areas  of  expertise, 
tend  to  be  overconfident  in  their  abilities,  and  overlook  the  less  relevant  matters  of  a 
subject  more  than  novices.  Additionally,  Hinds  (1999)  found  that  experts  may  not  be  as 
good  at  predicting  as  novices.  Furthermore,  Chi  (2006)  discusses  the  potentially  greatest 
limitation  of  experts  to  be  bias  which  tends  to  be  more  of  a  factor  in  experts  than  novices. 
Keeney  et  al.  (1991)  point  out  that  the  three  most  important  bias  considerations  when 
eliciting  information  from  experts  are  overconfidence,  anchoring,  and  availability.  They 
define  overconfidence  as  having  “more  certainty  in  judgments  than  is  appropriate”, 
anchoring  as  placing  too  much  “focus  on  an  initial  value”  followed  by  “insufficient 
adjustment”  to  additional  information,  and  availability  as  “overemphasis  of  events  that 
are  easily  imagined  or  recalled  (p.  199).”  However,  this  bias  was  mitigated  using  the 
nominal  group  technique  to  obtain  data  from  the  experts  in  this  study. 

Nominal  Group  Technique 

Although  a  variety  of  methods  used  to  obtain  dimension  data  from  experts  can  be 
used  such  as  brainstorming,  Delphi  technique,  voting,  interviews,  surveys,  the  nominal 
group  technique  was  selected  as  the  most  appropriate  method  in  this  study.  Reasons  for 
this  choice  include  the  ability  of  nominal  group  technique  to  generate  the  highest  volume 
of  ideas  (Dunnette,  Campbell,  &  Jaastad,  1963),  reduce  expert  bias  (Culbert,  1968),  and 
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obtain  quality  data  given  a  limited  amount  of  time  (Delbeeq  &  Van  de  Ven,  1971). 
Additionally,  it  has  been  shown  to  be  a  superior  method  over  brainstorming  by  producing 
more  ideas  and  with  greater  variety  (Bartunek  &  Murninghan,  1984)  and  there  is  a 
smaller  chance  of  groupthink  because  non-dominating  contributors  get  to  voice  their 
inputs  to  the  facilitator  but  dominating  members  have  to  wait  their  turn  (Heldman,  2005). 
However,  the  group  interaction  that  occurs  during  brainstorming  may  serve  as  an 
appropriate  means  of  priming  the  members  prior  to  exercising  individual  effort  or  non¬ 
interacting  nominal  groups  (Dunnette  et  ah,  1963;  Keeney  et  ah,  1991). 

Although  nominal  group  technique  has  been  observed  to  “produce  high  quality 
alternatives,  more  accurate  decisions  on  structured  problems,  lower  costs,  stronger 
feelings  of  accomplishment,  more  implementation  attempts,  more  satisfaction,  and  fewer 
negative  socioemotional  behaviors  than  free-flowing  group  problem-solving  procedures” 
(Bartunek  et  ah,  1984,  p.  418),  a  few  limitations  have  also  been  noted  and  addressed.  For 
example,  Bartunek  et  al.  (1984)  pointed  out  the  potentially  stifling  structure  of  nominal 
group  technique  to  only  produce  one  solution,  the  inadequacy  to  reach  consensus  for 
poorly  defined  problems,  and  the  inability  to  reformulate  the  initial  problem  statement 
once  the  process  has  begun.  Thus,  they  suggest  that  nominal  group  leaders  should 
engage  in  additional  steps  of  reflection  during  the  process  to  ensure  the  complexities  of  a 
problem  statement  are  satisfactorily  addressed  by  the  group.  Another  potential  problem 
Bartunek  et  al.  (1984)  point  out  regarding  nominal  group  technique  may  be  lack  of 
familiarity  with  the  process  and  perhaps  preference  for  more  free  flowing  discussions 
similar  to  brainstorming.  However,  such  an  issue  can  be  overcome  by  prefacing  the 
process  with  a  well-communicated  description  of  what  nominal  group  technique  is  and 
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the  benefits  it  has  shown  over  other  teehniques  like  brainstorming  (Bartunek  et  ah,  1984). 
Aecordingly,  prior  to  its  implementation  in  this  study,  a  detailed  explanation  of  nominal 
group  teehnique  as  outlined  by  Delbeeq  et  al.  (1971)  was  communicated  to  the 
participants. 

Mixed  Methods  Approach 

From  inception  (Campbell  &  Fiske,  1959)  to  more  modern  applications  (Creswell, 
2002),  the  mixed  methods  approach  has  been  increasingly  popularized.  The  mixed 
method  approach  utilizes  both  quantitative  and  qualitative  approaches  during  the  phases 
of  research  in  which  an  integration  of  these  methods  is  beneficial  (Creswell,  2002).  As 
Creswell  (2002)  asserts,  the  mixed  methods  approach  has  been  recognized  for  its  ability 
to  synthesize  findings  acquired  from  different  data  sources  and  gain  additional  research 
insight  when  single  method  approaches  fall  short.  Creswell  (2002)  suggests  how  the 
mixed  methods  approach  is  often  easiest  to  implement  when  conducted  in  sequentially 
(e.g.  qualitative  followed  by  quantitative)  rather  than  concurrently  (e.g.  qualitative  and 
quantitative  simultaneously).  Moreover,  he  provides  examples  of  mixed  methods 
approaches  to  include  the  use  of  both  open  ended  observations  and  close  ended  measures 
to  converging  multiple  forms  and  sources  of  data  such  as  statistical  analyses  and 
descriptive  information  (Creswell,  2002).  Accordingly,  this  study  is  representative  of  the 
mixed  methods  approach  because  of  data  collection  obtained  through  focus  groups 
followed  by  surveys  to  develop  leadership  criteria  under  maximum  performance 
conditions.  Using  this  approach  was  beneficial  to  gain  insight  on  which  dimensions  were 
considered  most  important  to  leadership  effectiveness. 
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Ill,  Method 


A  mixed  methods  approaeh  was  employed  as  a  means  of  converging  data 
obtained  from  both  focus  groups  and  surveys.  More  specifically,  a  sequential  mixed 
methods  strategy  as  suggested  by  Creswell  (2002)  was  used  for  data  collection  such  that 
the  data  extracted  from  focus  groups  was  followed  by  a  survey  to  confirm  the 
identification  of  the  most  important  leadership  criteria  under  maximum  performance 
conditions. 

The  procedure  used  in  this  study  started  with  data  collection  from  instructors  who 
qualified  as  experts  and  thus  were  considered  most  capable  of  contributing  meaningful 
inputs  that  related  to  leadership  performance.  These  experts  were  sent  an  initial  survey  as 
a  means  of  preparing  their  minds  for  eventual  focus  group  participation.  The  focus 
groups  were  conducted  using  the  nominal  group  technique  which  served  to  mitigate 
potential  biases  and  obtain  inputs  that  could  be  used  to  formulate  the  structure  of  the 
proposed  maximum  performance  leadership  dimensions.  Once  the  data  from  each 
assessment  center  were  compiled,  each  focus  group  input  was  reviewed  by  three  trained 
subject  matter  experts  to  ultimately  determine  a  psychological  dimension  that 
appropriately  captured  each  input  provided.  The  result  of  this  review  was  a  list  of 
leadership  dimensions  that  was  used  to  create  a  second  survey  which  asked  instructors  to 
rate  the  importance  of  each  dimension  relative  to  a  student’s  performance.  Finally,  an 
exploratory  factor  analysis  was  conducted  on  the  top  ten  leadership  dimensions  for  each 
assessment  center  as  a  means  of  uncovering  the  leadership  effectiveness  criteria. 
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Setting 


Participants  for  this  study  were  offieers  in  the  United  States  Air  Foree  assigned  as 
instructors  at  Air  and  Space  Basic  Course  (ASBC)  and  Squadron  Officer  Sehool  (SOS) — 
both  of  whieh  are  developmental  assessment  eenters  designed  to  educate  and  improve  the 
leadership  of  offieers.  Each  school  caters  to  a  level  of  management  in  the  Air  Foree  that 
is  determined  by  years  of  experience  and  scope  of  job  responsibility.  ASBC  is  a  six  week 
long  course  intended  for  officers  with  less  than  one  year  time  in  service  while  SOS  is  a 
five  week  long  course  tailored  to  officers  who  have  at  least  four  years  to  seven  years  time 
in  serviee  with  similar  managerial  experience.  Thus,  entry  level  management  describes 
ASBC  students  while  those  enrolled  in  SOS  are  better  characterized  as  middle  level 
management.  Such  a  cutoff  point  is  similarly  assigned  in  the  study  of  eross-cultural 
pereeptions  of  middle  management  eonducted  by  Neelankavil,  Mathur,  and  Zhang 
(2000).  Students  arriving  at  both  ASBC  and  SOS  represent  a  wide  range  of  jobs 
spanning  pilots  and  navigators  to  developmental  engineers  and  seientists. 

Sample 

A  total  of  37  ASBC  instructors  and  33  SOS  instructors  participated  in  the  data 
eollection  proeedure.  ASBC  instruetors  represent  middle  level  managers  with  four  to 
seven  years  experienee  while  SOS  instruetors  represented  more  senior  level  officers  with 
nine  to  twelve  years  of  experience.  As  mentioned  previously,  these  instruetors  were 
considered  experts  given  their  training  and  experience  related  to  specific  job  relevant 
events  and  situations. 
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Identifying  Leadership  Effectiveness  Dimensions 


Focus  Group  Preparation 

Identifying  potential  dimensions  started  with  foeus  group  preparation.  Careful 
consideration  was  taken  to  ensure  that  potential  dimensions  were  necessarily  supported 
by  evidence  from  instructor  inputs.  Thus,  prior  to  the  focus  groups,  instructors  were  sent 
a  survey  asking  them  to  list  training  events  that  best  helped  them  decide  who  was  a  good 
(or  bad)  leader.  Next,  they  were  asked  to  list  the  specific  behaviors  that  occur  (or  do  not 
occur)  during  each  of  the  events  which  help  them  to  assess  the  proposed  behaviors.  Both 
sets  of  questions  were  open  ended  to  ensure  an  unrestricted  range  of  answers. 
Additionally,  instructors  were  given  extensive  training  of  how  each  exercise  works  and 
the  expected  behaviors  that  students  exhibit  when  performing  the  exercises.  Thus,  there 
was  reduced  risk  that  potential  behaviors  or  events  were  excluded  from  consideration. 

The  results  were  compiled  from  these  surveys  to  obtain  a  baseline  of  information  that 
could  familiarize  the  researchers  in  this  study  for  eventual  focus  group  discussions.  More 
importantly,  such  an  approach  served  as  a  way  to  prime  the  minds  of  the  experts  for  the 
focus  groups  and  has  been  expressed  as  a  beneficial  precursor  to  using  the  nominal  group 
technique  as  a  means  to  facilitate  consensus  in  focus  groups  (Dunnette  et  ah,  1963). 

Focus  Group  Process 

Focus  groups  were  chosen  as  a  venue  for  the  nominal  group  technique  to  gain 
consensus  given  the  limited  amount  of  time  of  instructors  and  their  availability  to  provide 
inputs.  To  facilitate  the  focus  groups,  the  nominal  group  technique  was  used  since  it  has 
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been  empirically  supported  (Dunnette,  1963)  and  has  been  described  as  one  of  the  best 
and  most  efficient  means  of  producing  quality  inputs  in  a  group  setting  (Bartunek,  1984). 
Using  the  procedures  outlined  by  Delbecq  and  Van  de  Ven  (1971),  the  focus  groups  were 
formed  in  groups  of  4-8  instructors.  Six  focus  groups  were  formed  for  ASBC  using  37 
instructors  and  four  groups  were  formed  for  SOS  using  33  instructors.  During  the  focus 
groups,  participants  were  asked  to:  1)  list  as  many  characteristics  of  a  low  performing 
student  as  they  could  and  2)  similarly  for  a  high  performing  student.  For  each  question, 
the  group  was  given  enough  time  (typically  20-30  minutes)  of  silence  to  write  one  input 
on  a  single  piece  of  paper  that  they  felt  answered  the  question.  Moreover,  it  was 
emphasized  to  the  instructors  that  the  characteristics  must  be  tied  to  performance.  At  the 
end  of  this  segment,  inputs  were  solicited  by  each  instructor  one  at  a  time.  Inputs  were 
systematically  provided  by  rotating  between  each  member  of  the  group.  Each  input  was 
visibly  displayed  by  posting  it  on  an  easel.  Additionally,  each  input  was  subject  to  the 
debate  of  the  team.  Inputs  that  were  rejected  were  also  visibly  posted  for  the  group’s 
reference.  If  a  duplicate  input  was  shared  by  another  member  of  the  team,  both  inputs 
would  be  combined  and  similarly  displayed  on  the  easel.  As  a  means  of  categorizing 
conceptually  similar  inputs,  headings  were  debated  and  established  by  the  group  to 
identify  the  ones  that  most  appropriately  captured  the  inputs.  When  a  question  was 
sufficiently  saturated  with  inputs  or  when  time  warranted  each  session,  the  next  question 
would  be  introduced.  The  final  result  was  a  list  of  behaviors  deemed  critical  to  the 
performance  of  a  student. 
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Using  the  nominal  group  technique  elicited  the  best  inputs  because  it  provided  a 
group  of  experts  an  opportunity  to  write  down  their  answers  before  being  asked  to 
provide  inputs  one-by-one  resulting  in  a  wide  range  of  expert  inputs.  Once  inputs  were 
provided,  all  of  the  instructors  were  given  the  chance  to  provide  feedback  regarding  each 
input.  Moreover,  such  a  process  encouraged  experts  to  contribute  their  highest  effort 
since  each  member  was  able  to  directly  observe  how  their  specific  input  impacted  the 
quality  of  the  study  (Keeney  et  ah,  1991).  The  result  of  this  process  was  a  list  of 
characteristics  considered  to  be  most  relevant  to  the  job.  Additionally,  non-dominating 
personalities  were  able  to  voice  their  inputs  just  as  much  as  more  dominating  members. 
This  particular  feature  of  the  nominal  group  technique  was  critical  to  reduce  bias  and 
counteract  groupthink  (Heldman,  2005). 

Focus  Group  Data  Analysis 

The  focus  group  data  analysis  was  achieved  in  three  phases.  The  first  phase 
involved  compiling  the  focus  group  data  for  analysis.  A  total  of  347  ASBC  inputs  (174 
high  performer  and  173  low  performer)  and  178  SOS  inputs  (82  high  performer  and  96 
low  performer)  were  inputted  into  a  database.  Additionally,  the  headings  established 
during  the  focus  groups  for  each  set  of  inputs  were  similarly  entered  into  the  database. 
The  second  phase  consisted  of  determining  appropriate  dimensions  that  encompassed  the 
idea  behind  each  instructor’s  input.  This  was  achieved  using  three  trained  subject  matter 
experts  (two  industrial-organizational  PhD  psychologists  and  one  management  PhD)  to 
analyze  the  data.  Each  input  was  reviewed  and  assigned  potential  dimensions  that 
encapsulated  the  comment.  To  reduce  potential  availability  bias,  the  previously 
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established  focus  group  headings  were  hidden  from  view  during  this  phase  of  the 
analysis.  The  third  phase  consisted  of  selecting  a  final  dimension  by  comparing  the  focus 
group  headings  with  the  potential  dimensions  identified  during  phase  two.  These  final 
dimensions  were  selected  according  to  their  ability  to  most  closely  resemble  the  focus 
group  headings  (and  therefore  set  of  inputs)  to  ensure  alignment  with  the  intention  of  the 
focus  group  inputs  (the  context).  The  finalized  listing  of  dimensions  was  then  collapsed 
further  to  combine  the  multiple  instances  in  which  identical  dimensions  emerged.  The 
result  of  this  phase  determined  the  number  of  dimensions  derived  from  the  focus  group 
inputs.  A  total  of  63  dimensions  emerged  from  the  ASBC  focus  group  inputs  and  97 
dimensions  were  formulated  for  SOS. 

A  second  survey  was  then  assembled  and  disseminated  using  the  dimensions 
describing  ASBC  and  SOS.  Specifically,  this  survey  consisted  of  the  total  number  of 
dimensions  along  with  a  brief  characteristic  that  concisely  described  each  dimension. 
These  characteristics  were  derived  from  items  used  to  measure  each  dimension.  The 
instructors  were  asked  to  rate  on  a  scale  of  1-10  how  important  they  thought  each 
characteristic  was  to  determining  a  high  performer.  Similarly,  for  negative  dimensions, 
e.g.  counterproductive  work  behavior,  the  instructors  were  asked  to  rate  on  a  scale  of  1- 
10  how  destructive  they  thought  each  characteristic  was  to  the  performance  of  a  trainee. 
Finally,  the  results  from  this  survey  were  inputted  into  a  database  to  identify  the  overall 
relative  importance  of  each  dimension. 
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Identifying  Leadership  Effectiveness  Criteria 

To  identify  leadership  effectiveness  criteria,  the  mean  ratings  were  used  to  rank 
the  dimensions  in  terms  of  importance  of  behaviors  to  performance  as  viewed  by  the 
instructors.  Once  a  complete  listing  of  dimensions  for  ASBC  and  SOS  were  rank 
ordered,  the  top  ten  dimensions  from  each  assessment  were  compared  by  conducting  an 
exploratory  factor  analysis  and  interpreting  the  results  using  a  varimax  rotation.  Only 
substantively  important  factors  were  retained  based  on  the  Kaiser  (1960)  criterion  of 
retaining  factors  with  eigenvalues  greater  than  one.  After  suppressing  results  less  than  .4 
as  proposed  by  Stevens  (2002),  two  dimensions  from  SOS  were  discarded  due  to 
persisting  cross-loading.  The  factor  analysis  yielded  three  factors  for  ASBC  and  three 
factors  for  SOS  . 
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IV:  Analysis  and  Results 


As  previously  discussed,  rigorous  dimension  development  was  necessary  to 
identify  appropriate  leadership  effectiveness  criteria.  The  result  was  a  list  of  dimensions 
derived  from  expert  inputs  consisting  of  leadership  behaviors  deemed  relevant  to 
maximum  performance.  Although  ASBC  yielded  63  dimensions  and  SOS  had  97,  a 
cutoff  was  made  between  the  two  assessment  centers  in  order  to  compare  the  most 
significant  differences.  Accordingly,  the  top  ten  dimensions  ranked  by  mean  importance 
were  used  for  further  analysis  to  determine  these  meaningful  differences.  Below,  Table 
1  highlights  the  highest  rated  dimensions  and  associated  characteristics  for  ASBC  and 
Table  2  displays  the  same  information  for  SOS: 


Dimensions: 

Characteristics: 

1. 

Counterproductive  Work 
Behavior 

1. 

Exhibits  active  negative  behaviors 

2. 

Integrity 

2. 

Conveys  a  clear  sense  of  integrity 

3. 

Negative  Affect 

3. 

Degree  to  which  student  exhibits  a  negative  attitude 

4. 

Honesty 

4. 

Willingness  of  the  student  to  be  honest 

5. 

Mastery  Orientation 

5. 

Desire  to  do  the  best  job  possible  in  all  areas 

6. 

Idealized  Influence 

6. 

Ability  to  act  as  strong  role  model  and  make  others  want 
to  follow 

7. 

Conscientiousness 

7. 

Attention  to  detail;  awareness  of  what  needs  to  be  done 
and  prepares  accordingly 

8. 

Engagement 

8. 

Willingness  to  actively  participate  and  remain  engaged  in 
activities 

9. 

Verbal  Communication 

9. 

Ability  to  effectively  communicate  verbally 

10. 

Effort 

10. 

Lack  of  effort  applied 

Table  1:  Top  Ten  Dimensions  and  Associated  Characteristics  for  ASBC 
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Dimensions: 

Characteristics: 

1. 

Followership 

1. 

Ability  of  student  to  follow  as  well  as  lead 

2. 

Feedback  Acceptance 

2. 

Willingness  to  accept  feedback 

3. 

Meta  Cognitive 
Prompting 

3. 

Ability  to  provide  strong  and  clear  direction 

4. 

Trustworthy 

4. 

Degree  to  which  student  is  trustworthy 

5. 

Selflessness 

5. 

Degree  to  which  student  is  selfless 

6. 

Cognitive  Adaptability 

6. 

Inability  to  adapt  to  new  information  or  environmental 
factors 

7. 

Developing  Team 
Members 

7. 

A  team  player;  willing  to  encourage  and  support 
teammates  to  increase  their  effectiveness 

8. 

Introspection 

8. 

Ability  to  self-critique  to  understand  what  is  needed  to 
be  worked  on 

9. 

Honesty 

9. 

Willingness  of  the  student  to  be  honest 

10. 

Verbal  Communication 

10. 

Ability  to  effectively  communicate  verbally 

Table  2:  Top  Ten  Dimensions  and  Associated  Characteristics  for  SOS 


Exploratory  Factor  Analysis 

An  exploratory  factor  analysis  using  SPSS  was  conducted  on  the  mean 
importance  of  dimensions  as  rated  by  each  instructor.  However,  to  ensure  there  was 
support  for  the  use  of  an  exploratory  factor  analysis  for  uncovering  leadership  criteria,  a 
series  of  statistical  tests  were  conducted.  Sampling  adequacy  was  determined  using  the 
Kaiser-Meyer-Olkin  (KMO)  test  which  resulted  in  values  of  .635  for  ASBC  and  .690  for 
SOS  (i.e.  well  above  the  acceptable  limit  of  .5  (Field,  2009)).  Bartlett’s  test  was  used  to 
ensure  sufficient  correlations  between  dimensions  existed  and  found  that  each  dimension 
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was  less  than  .05  for  each  assessment  center  (i.e.  .000  for  both  ASBC  and  SOS) 
indicating  the  correlations  between  each  variable  were  significantly  different  from  zero. 
Moreover,  the  determinant  of  each  correlation  matrix  was  greater  than  .00001  (i.e.  .009 
for  ASBC  and  .004  for  SOS)  and  therefore  indicated  no  severe  multicollinearity  (Field, 
2009).  Based  on  the  tests  of  sampling  adequacy,  correlations  between  dimensions,  and 
severe  multicollinearity,  a  meaningful  comparison  of  the  factors  retained  for  both  ASBC 
and  SOS  was  possible. 

Conducting  the  exploratory  factor  analysis,  tens  factors  emerged  for  both  ASBC 
and  SOS.  However,  three  factors  were  retained  for  ASBC  and  SOS  using  the  Kaiser 
(1960)  criterion  of  analyzing  factors  with  eigenvalues  greater  than  one.  Moreover,  there 
was  a  significant  drop  in  the  eigenvalue  from  the  third  factor  to  subsequent  factors.  Next, 
to  compare  leadership  criteria  between  ASBC  and  SOS,  the  factor  loading  tables  for  each 
assessment  center  were  observed  to  identify  which  dimensions  tied  to  each  factor  and  to 
determine  if  there  were  any  significant  differences.  Table  3  and  Table  4  show  the 
ASBC  and  SOS  factor  loadings,  respectively,  of  each  dimension  after  suppressing  results 
less  than  .4  to  facilitate  interpretation  of  factor  loadings  and  after  subjecting  these  data  to 
a  varimax  rotation.  Note  that  the  ASBC  factor  loadings  were  achieved  using  only  27  of 
37  surveys  due  to  lack  of  returned  surveys  or  incomplete  surveys. 
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ASBC  (n=27) 


Factor 

Dimension 

1 

2 

3 

Conscientiousness 

.874 

Idealized  Influence 

.832 

Honesty 

.848 

Integrity 

.750 

Mastery  Orientation 

.616 

Verbal  Communication 

.769 

Engagement 

.830 

Effort 

.661 

Counterproductive  Work  Behavior 

.752 

Negative  Affect 

.803 

Table  3:  Factor  Loadings  for  ASBC 


SOS  (11=33) 


Factor 

Dimension 

1 

2 

3 

Honesty 

.539 

DevelopingTeam  Members 

.800 

Verbal  Communication 

.653 

Selflessness 

.582 

Trustworthiness 

.920 

Followership 

.791 

Feedback  Acceptance 

.804 

Introspection 

.874 

Metacognitive  Prompting 

.403 

.676 

Cognitive  Adaptability 

.801 

Table  4:  Factor  Loadings  for  SOS 
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Together,  the  three  retained  ASBC  factors  explained  65.813%  of  the  variance 
while  the  three  SOS  factors  explained  69.883  %  of  the  variance.  Table  5  represents  the 
%  variance  explained  per  factor  for  ASBC  and  Table  6  displays  this  information  for 
SOS.  Note  that  the  fourth  factor  is  shown  only  to  highlight  the  significant  drop  in 
eigenvalues  that  occurred  (and  within  each  subsequent  factor)  for  both  ASBC  and  SOS. 


Factor 

Eigenvalue 

%  of  Variance 

Cumulative  % 

1 

3.799 

37.994 

37.994 

2 

1.448 

14.477 

52.471 

3 

1.334 

13.342 

65.813 

4 

.868 

8.680 

74.494 

Table  5:  %  Variance  Explained  Per  Factor  for  ASBC 


Factor 

Eigenvalue 

%  of  Variaixe 

Cumulative  % 

1 

4.380 

43.805 

43.805 

2 

1.412 

14.118 

57.923 

3 

1.196 

11.961 

69.883 

4 

.848 

8.481 

78.364 

Table  6:  %  Variance  Explained  Per  Factor  for  SOS 


For  ASBC,  the  dimensions  that  tied  together  will  be  discussed  in  more  detail  but 
indicated  that  factor  1  represents  personality,  factor  2  represents  effort,  and  factor  3 
represents  attitude.  To  more  accurately  describe  the  factors  that  emerged  for  SOS,  two 
dimensions  were  discarded  (i.e.  honesty  and  metacognitive  prompting)  due  to  cross 
loading  once  results  less  than  .4  were  suppressed.  Thus,  upon  further  analysis  of  the  SOS 
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factors,  factor  1  seemed  to  more  appropriately  represent  trust,  factor  2  represented 
accommodation,  and  factor  3  represented  adaptability.  Taken  together,  the  leadership 
effectiveness  criteria  for  the  entry  level  managers  represented  by  ASBC  suggested  that 
“leadership  of  self’  is  most  important  while  “leadership  of  team”  seems  to  best  describe 
middle  management  leadership  effeetiveness  criteria  represented  by  SOS. 
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V:  Discussion  and  Conclusions 


Discussion 

Many  researchers  have  contributed  studies  attempting  to  advance  the 
understanding  of  typical  and  maximum  performance,  but  none  have  empirically  or 
theoretically  explored  the  leadership  criteria  elicited  by  maximum  performance 
conditions.  Moreover,  none  have  attempted  to  distinguish  between  leadership  criteria 
under  maximum  performance  conditions  as  it  relates  to  both  entry  and  middle  levels  of 
management.  Accordingly,  this  effort  was  designed  to  identify  the  leadership  criteria  that 
relates  to  both  entry  and  middle  level  management. 

Using  the  robust  process  outlined  by  Thornton  and  Rupp  (2006)  was  necessary  to 
accurately  identify  dimensions  relevant  to  ASBC  and  SOS  leadership  performance. 

These  data  were  elicited  from  qualified  subject  matter  experts  using  nominal  group 
technique  and  consisted  of  job  content  deemed  to  be  an  accurate  representation  of 
leadership  effectiveness  dimensions.  The  dimensions  that  emerged  from  the  data  were 
analyzed  using  exploratory  factor  analysis  which  revealed  three  factors  for  ASBC  and 
three  factors  for  SOS.  In  particular,  researchers  should  take  note  of  the  top  rated 
dimensions  for  ASBC  and  SOS  and  how  the  factors  describe  each  level  of  management. 
In  doing  so,  the  leadership  criteria  for  both  entry  and  middle  level  managers  exposed  to 
maximum  performance  conditions  are  observed. 
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Entry  Level  Management  Leadership  Criteria 


Personality 


Conscientiousness 


Idealized  Influenee 


Honesty 


Integrity 


Mastery  Orientation 


Verbal  Communication 


Table  7:  ASBC  Factor  1 


Six  of  the  top  ten  ASBC  dimensions  are  tied  to  factor  1  as  seen  in  Table  7.  A 
eomponent  of  the  well-supported  Big  Five  personality  model,  conscientiousness, 
emerged  as  one  of  the  most  important  factors  to  performance  for  entry  level  managers 
engaged  in  maximum  performanee.  Perhaps  unsurprising,  this  personality  trait  has  been 
observed  to  be  one  of  the  highest  contributors  to  “can  do”  performance,  as  elicited  by 
maximum  performanee  (Barrick,  Mount,  &  Strauss,  1993;  Borman,  White,  Pulakos,  & 
Oppler,  1991)). 

Additionally,  there  is  a  trait  component  of  leadership  that  resides  in  the 
transformational  leadership  model  deseribed  as  idealized  influence  (Northouse,  2001). 
Of  the  “Four  I’s”  of  transformational  leadership,  idealized  influenee  is  deseribed  as  the 
charismatie  and  role  model  eharacteristie  of  leadership  and  is  what  entices  individuals  to 
follow  the  leader  (Bono  &  Judge,  2004;  Bass  &  Steidlmeier,  1999).  Moreover,  there  is  a 
moral  dimension  encompassed  by  idealized  influenee  whereby  honesty  and  integrity  are 
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considered  signifieant  eontributors  (Brown,  Trevino,  &  Harrison,  2005;  Bass  & 
Steidlmeier,  1999).  In  other  words,  the  traits  of  honesty  and  integrity  are  tied  together 
and  significantly  contribute  to  idealized  influence,  or  the  trait  leadership  component  of 
the  transformational  leadership  model.  Thus,  it  should  not  be  surprising  the  traits  of 
honesty,  integrity,  and  idealized  influence  clustered  to  the  same  factor.  Moreover,  the 
fact  that  idealized  influence  (and  thus,  honesty  and  integrity)  emerged  as  a  top  dimension 
of  maximum  performance  further  supports  previous  findings  that  transformational 
leadership  is  most  important  to  maximum  performance  compared  to  typieal  performanee 
(Lim  et  ah,  2004;  Ployhart  et  ah,  2001).  Additionally,  there  is  evidenee  that 
transformation  leadership  has  been  linked  to  personality  dimensions  within  the  Big  Five 
framework  (Lim  et  ah,  2004;  Ployhart  et  ah,  2001). 

London  and  Smither  (2002)  define  mastery  orientation  as  when  an  individual’s 
“attentional  focus  is  on  developing  competenee.  These  learners  want  to  aequire 
knowledge  and  skill  until  they  reaeh  a  level  of  mastery  that  reflects  a  deep  (expert) 
understanding,  and  they  view  feedback  about  skill  deficits  as  an  opportunity  for 
improvement”  (London  et  ah,  2002,  p.83).  Thus,  it  makes  intuitive  sense  that  mastery 
oriented  entry  level  managers  exposed  to  the  maximum  performance  eonditions  of 
ASBC’s  developmental  assessment  center  would  feel  a  strong  sense  to  develop 
competency  because  by  definition,  the  experience  and  knowledge  of  entry  level  managers 
is  lacking,  and  therefore,  the  desire  to  gain  competeney  is  intensified. 

Additionally,  mastery  orientation  is  referred  to  as  trait  learning  orientation  due  to 
the  consistent  finding  that  it  stems  from  a  stable  personal  disposition,  or  trait  derived 
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from  one’s  personality  (Ward,  Rogers,  Byrne,  &  Masterson,  2004).  Moreover,  the 
eonditions  that  Ward  et  al.  (2004)  established  in  their  experiment  were  representative  of 
maximum  performance  in  that  subjects  were  aware  they  were  being  evaluated,  maximum 
effort  was  expected  and  exerted,  and  the  evaluative  period  was  over  a  short  duration  (30 
minutes).  In  other  words,  mastery  orientation — or  trait  learning  orientation — is  trait 
based  and  personality  driven  and  has  been  observed  to  remain  stable  across  performance 
and  learning  based  situations  as  elicited  by  assessment  centers  like  ASBC. 

Finally,  it  should  not  be  surprising  to  see  verbal  communication  as  essential  to 
leadership  effectiveness.  Leadership  requires  the  ability  to  communicate  and  influence 
others  to  achieve  organizational  goals.  Accordingly,  verbal  communication  was 
considered  to  be  among  the  top  criteria  for  leadership  effectiveness  for  both  entry  and 
middle  level  managers.  As  a  leadership  dimension,  verbal  communication  has  also  been 
referred  to  as  speech  fluency,  or  an  ability  characteristic  that  is  trait  based  (Northouse, 
2001). 

Taken  together,  conscientiousness,  honesty,  idealized  influence,  integrity,  mastery 
orientation,  and  verbal  communication  suggests  that  entry  level  managers  exposed  to 
maximum  performance  conditions  can  attribute  a  significant  portion  of  their  leadership 
effectiveness  to  traits,  dispositional  characteristics,  and  ability  characteristics.  Moreover, 
traits,  dispositional  and  ability  characteristics  are  said  to  comprise  an  individual’s 
personality  (McCrae  &  John,  1992;  Tellegen,  1991).  In  other  words,  personality  emerged 
as  one  of  the  most  important  components  of  performance  under  maximum  performance 
conditions  for  entry  level  managers. 
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Effort 


Engagement 

Effort 

Tables:  ASBC  Factor  2 

The  seeond  faetor  that  emerged  at  ASBC  eonsisted  of  engagement  and  effort  as 
seen  in  Table  8.  One  of  the  eonditions  of  maximum  performanee  as  noted  by  Saekett  et 
al.  (1988)  is  that  individuals  must  exert  effort.  Moreover,  they  suggest  that  if  everyone 
tries  hard,  then  effort  is  not  a  eontributor  to  the  varianee.  However,  this  dimension 
suggests  that  those  who  try  hard  (i.e.  exhibit  high  amount  of  effort)  will,  on  average,  be 
more  effeetive  than  those  who  do  not  (Eisher  &  Eord,  1998).  Aeeordingly,  the  essenee  of 
engagement  is  a  persistenee  of  attentiveness  and  effort  (Skinner  &  Belmont,  1993)  and 
therefore  indieates  that  as  a  whole,  effort  is  a  signifieant  eriterion  to  leadership 
effeetiveness  for  entry  level  managers. 

Attitude 

Counterproduetive  Work  Behavior 
Negative  Affeet 

Table  9:  ASBC  Factor  3 

Table  9  highlights  the  third  faetor  for  ASBC  in  whieh  counterproductive  work 
behavior  and  negative  affect  tied  together.  Counterproductive  work  behavior  is  described 


36 


as  a  realized  behavior  that  runs  contrary  to  the  goals  of  an  organization  and  that  may 
manifest  itself  through  unsafe  behavior,  misuse  of  time  and  resources,  and  poor  quality 
work  (Sackett,  2002).  Negative  affect  is  defined  as  the  unpleasureable  engagement  in 
activities  that  stems  from  a  variety  of  aversive  mood  states  (Watson,  Clark,  &  Tellegen, 
1988)  which  can  also  result  in  aversive  behaviors  (Brown,  Chorpita,  &  Barlow,  1998). 
Thus,  the  link  between  counterproductive  work  behaviors  and  negative  affect  is  a  logical 
one  and  makes  sense  that  these  dimensions  tied  to  the  same  factor.  Considered 
collectively,  this  factor  suggests  a  third  criterion  of  leadership  effectiveness  to  be  attitude. 
More  specifically,  this  finding  suggests  that  a  bad  attitude  significantly  detracts  from  the 
effectiveness  of  a  junior  leader  which  makes  sense  when  considering  the  observation  that 
“attitudes  mediate  the  effect  of  the  external  stimulus  world  on  overt  behavior”  (Organ  & 
Bateman,  1991,  p.l80).  Thus,  in  a  leadership  context,  a  bad  attitude  negatively  affects 
the  disposition  of  others  which  in  turn,  produces  undesirable  behaviors  indicative  of  poor 
leadership  effectiveness. 

Middle  Level  Management  Leadership  Criteria 

Similar  to  ASBC,  SOS  had  three  factors  associated  with  the  top  rated  dimensions 
despite  discarding  two  dimensions  due  to  cross  loading  as  mentioned  previously  (i.e. 
honesty  and  metacognitive  prompting).  A  discussion  of  the  SOS  factors  is  presented. 

In  the  AIB  model  of  trust  (Mayer,  Davis,  &  Shoorman,  1995),  the  propensity  to 
trust  depends  on  the  trustee’s  ability,  integrity,  and  benevolence.  Moreover,  ability, 
integrity,  and  benevolence  are  considered  the  factors  of  perceived  trustworthiness.  This 
concept  of  trust  is  considered  the  “lubricant”  for  high  performing  teams  such  that  in  its 
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absence,  the  pace  of  team  performance  grinds  to  a  halt  (Mayer  et  ah,  1995).  Thus,  when 
viewed  through  the  AIB  framework,  the  factor  of  “trust”,  as  displayed  in  Table  10,  is 
explained  by  considering  developing  team  members  and  verbal  communication  as 
abilities,  selflessness  as  an  act  of  benevolence,  and  trustworthiness  as  the 
operationalization  of  ability,  integrity,  and  benevolence. 


Trust 


Developing  Team  Members 


Verbal  Communication 


Selflessness 


Trustworthiness 


Table  10:  SOS  Factor  1 


Zacarro,  Rittman,  and  Marks  (2001)  suggest  that  developing  team  members  is  the 
ability  to  establish  appropriate  mental  models  that  the  team  can  use  to  process 
information  and  solve  problems  accordingly.  The  successful  transfer  of  an  appropriate 
mental  model  depends  on  the  leader’s  ability  to  communicate  the  information  used  to 
form  these  mental  models.  Thus,  it  should  not  be  surprising  that  the  dimensions  of 
developing  team  members  and  verbal  communication  both  tied  together  as  essential 
abilities. 
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Selflessness  ean  be  defined  as  the  willingness  to  make  saerifiees  for  the  overall 
benefit  of  the  team.  For  example,  the  willingness  to  put  in  more  time  than  usual  during 
peak  periods  of  labor  and  at  the  expense  of  personal  time  ean  be  eonsidered  exemplifying 
selflessness.  In  this  sense,  the  individual  demonstrating  selflessness  is  aeting  out  of 
benevolenee  for  the  benefit  of  the  team. 

Taken  together,  the  SOS  faetor  of  trust  is  understood  in  the  eontext  of  the  Mayer 
et  al.  (1995)  AIB  model  of  trust  whieh  suggests  ability  (developing  team  members,  verbal 
eommunieation),  integrity,  and  benevolenee  (selflessness)  are  integral  eomponents  of 
trustworthiness  which  is  the  “lubricant”  of  team  effectiveness  (Mayer  et  ah,  1995).  Thus, 
this  factor  of  trust  was  identified  as  one  of  the  most  important  contributors  to  middle 
level  management  leadership  effectiveness. 

The  second  factor  that  emerged  to  describe  leadership  effectiveness  criteria 
consisted  of  followership,  introspection,  and  feedback  acceptance.  The  concept  of 
followership  has  come  a  long  way  since  the  traditional  “passive  subordinate”  paradigm 
and  has  more  recently  been  considered  an  important  component  of  leadership.  This  more 
contemporary  view  acknowledges  the  importance  of  followers  to  organizational 
performance  (Kelley,  2008),  the  role  that  leaders  and  followers  play  as  highlighted  by  the 
leader-membership  exchange  theory  (Graen  &  Uhl-Bien,  1995),  and  how  both  roles  are 
essential  to  an  organization  such  that  there  are  times  when  leaders  need  to  be  good 
followers,  and  vice-versa  (Litzinger  &  Schaefer,  1982).  The  other  dimensions  of  this 
factor,  introspection  and  feedback  acceptance,  can  be  better  understood  in  terms  of  the 
feedback  orientation  framework  (London  &  Smither,  2002).  In  this  framework,  seeking. 
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processing,  and  accepting  the  feedback  are  seen  as  parts  of  a  behavioral  changing  process 
to  increase  self-awareness  (i.e.  introspection  (Boring,  1953))  and  performance.  Thus,  the 
importance  of  a  leader  to  be  able  to  enaet  followership  and  change  behavior  aecording  to 
feedback  suggests  that  accommodation  is  a  necessary  criterion  for  leadership 
effectiveness  and  is  shown  in  Table  11.  It  is  this  accommodation  that  enables  the  leader 
to  be  open  to  conceding  to  others,  aware  of  one’s  strengths  and  weaknesses,  and  receptive 
to  feedback  in  order  to  beeome  a  more  effective  leader. 

Accommodation 

Followership 
Feedback  Acceptance 
Introspection 

Table  11:  SOS  Factor  2 

The  third  factor  consisted  of  only  one  dimension,  cognitive  adaptability,  which 
reflects  the  ability  of  an  individual  to  adapt  to  environmental  ehanges  and  overcome 
constraints  (Canas,  Quesada,  Antoli,  &  Fajardo,  2003).  Given  the  dynamic  environments 
of  organizations,  it  is  little  wonder  the  ability  to  adapt  to  the  nuances  of  constraints, 
uncertainty,  and  resistance  encountered  by  middle  level  managers  is  considered  to  be 
among  the  most  important  dimensions  to  possess.  Thus,  as  Table  12  highlights,  the 
concept  of  adaptability  seemed  to  be  an  appropriate  criterion  of  leadership  effectiveness 
for  middle  managers. 
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Adaptability 

Cognitive  Adaptability 

Table  12:  SOS  Factor  3 

When  eonsidering  the  ASBC  top  ten  leadership  dimensions,  the  three  faetors  that 
emerged  ean  be  deseribed  as  personality,  effort,  and  attitude.  Personality  was  comprised 
of  conscientiousness,  honesty,  idealized  influence,  integrity,  mastery  orientation,  and 
verbal  communication.  Collectively,  these  traits,  dispositional  characteristics,  and  ability 
characteristics  were  representative  of  personality.  Examined  further,  these  personality 
traits  suggest  that  this  entry  level  manager  factor  is  operationalized  within  an  individual’s 
self-control.  In  other  words,  effectiveness  within  this  factor  seems  to  be  achieved  when 
the  individual  chooses  to  exercise  self-control  measures  such  as  trying  to  maintain 
awareness  or  conscientiousness,  upholding  a  higher  moral  standard  by  being  honest  and 
acting  with  integrity,  desiring  to  gain  competency,  and  exhibiting  fluency  in  verbal 
communication.  The  effort  factor  suggested  that  those  who  exert  a  high  amount  of  effort 
and  engagement  will  be  exhibiting  an  important  leadership  effectiveness  criterion  for 
entry  level  managers.  That  is,  exerting  a  high  level  of  effort  is  important  for  entry  level 
managers.  Similar  to  the  personality  factor,  effort  is  also  operationalized  in  the 
individual  domain  and  can  therefore  be  considered  a  stand-alone  factor.  An  individual 
controls  the  level  of  effort  exerted  and  is  not  dependent  on  the  influence  of  an  external 
factor  to  exist.  Finally,  the  factor  of  attitude  consisted  of  counterproductive  work 
behavior  and  negative  affect  whereby  attitude  has  been  found  to  drive  the  interaction 
(Organ  &  Bateman,  1991)  and  thus  suggesting  that  attitude  was  important  to  leader 
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effectiveness  for  entry  level  managers.  Once  again,  attitude  is  within  the  control  of  an 
individual  and  is  not  contingent  on  the  interaction  with  an  external  stimulus.  An 
individual  can  choose  to  have  a  good  or  bad  attitude  regardless  of  the  individual’s 
external  environment.  Considering  the  significant  emphasis  on  the  individual  domain, 
the  factors  of  personality,  effort,  and  attitude  suggest  that  “leadership  of  self’  is  the 
overarching  construct  describing  entry  level  manager  leadership  effectiveness. 

Interesting  to  note,  the  majority  of  the  leadership  criteria  for  middle  level 
managers  corresponded  to  the  interaction  between  the  individual  and  external  stimuli — a 
marked  contrast  to  the  individual  domain  that  composed  the  “leadership  of  self’  criteria 
of  entry  level  managers.  Trust,  accommodation,  and  adaptability  emerged  as  the  most 
important  leadership  criteria  at  the  middle  management  level.  Trustworthiness,  verbal 
communication,  selflessness,  developing  team  members,  followership,  feedback 
acceptance,  and  cognitive  adaptability  are  all  dimensions  that  encapsulate  the  interaction 
of  the  individual  and  an  external  influence  (others  or  the  environment).  For  an  individual 
to  be  trustworthy,  he  or  she  must  be  entrusted  by  another  (the  trustee),  and  thus  an 
interaction  between  individual  and  another  individual  or  a  group  is  necessary. 
Additionally,  the  accommodation  of  others  referred  to  an  individual’s  openness  to 
concede  to  others  as  determined  through  awareness  of  one’s  strengths  and  weaknesses, 
and  receptivity  to  feedback  from  others.  Moreover,  the  dimensions  of  followership  and 
feedback  acceptance  each  require  reaction  to  the  effects  of  others.  Finally,  cognitive 
adaptability  was  described  as  an  individual’s  ability  to  adapt  to  changes  in  the  external 
environment.  Aside  from  introspection,  which  centers  on  an  individual’s  awareness  of 
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self,  these  results  suggest  the  majority  of  the  most  important  middle  level  manager 
dimensions  are  contingent  on  the  interaction  between  the  individual  and  an  external 
influence.  Accordingly,  considering  the  heavy  emphasis  on  the  interaction  of  the 
individual  domain  with  others,  “leadership  of  team”  seems  to  appropriately  describe  the 
leadership  effectiveness  for  middle  level  managers.  These  findings  are  summarized  in 
Table  13. 


Entry  Level  Managers 
"Leadership  of  Self" 

Personality 

Effort 

Attitude 

Middle  Level  Managers 
"Leadership  of  Team  " 

Trust 

Accommodation 

Adaptability 

Table  13:  Leadership  Effectiveness  Criteria  for  Entry  and  Middle  Level  Managers 
Conclusion 

An  empirical  study  was  conducted  to  shed  light  on  the  study  of  maximum 
performance  of  entry  and  middle  levels  of  management.  It  appears  this  study  suggests  an 
alternative  leadership  skills  structure  for  entry  and  middle  level  managers  when 
compared  to  the  strataplex  of  leadership  model  (Mumford,  Campion,  &  Morgeson, 

2007). 

The  strataplex  model  was  built  using  the  0*NET  leadership  skills  database 
consisting  of  skills  and  job  analyses  developed  by  the  U.S.  Department  of  Labor  as  a 
means  of  describing  work  across  different  job  domains  (Mumford  et  ah,  2007). 

Moreover,  the  strataplex  was  intended  to  compare  the  levels  of  organizational  leadership 
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(entry,  middle,  senior)  and  how  leadership  skills  (cognitive,  interpersonal,  business,  and 
strategic)  within  each  level  varied.  Cognitive  skills  referred  to  information  collecting, 
processing,  disseminating,  and  learning.  Examples  of  these  skills  were  verbal  and  written 
communication,  active  listening,  reading  comprehension,  active  learning,  and  critical 
thinking.  Interpersonal  skills  were  described  as  the  ability  to  interact  and  socialize  with 
others  and  included  examples  such  as  social  judgment,  social  complexity  and 
differentiation,  human  relation  skills,  social  perceptiveness,  coordination,  negotiation, 
and  persuasion.  Business  skills  were  essentially  the  KSAs  that  specifically  related  to  the 
job  context  and  included  management  of  material  and  resources,  procuring  and  allocating 
equipment,  technology,  personnel,  and  financial  matters.  Finally,  strategic  skills  related 
to  systems  thinking  as  a  way  to  understand  complexity  and  included  visioning,  systems 
perception,  identification  of  downstream  consequences,  environmental  scanning, 
identification  of  key  problems  and  root  causes,  and  evaluation  of  alternative  courses  of 
action  (Mumford  et  ah,  2007). 

Since  the  strataplex  model  focused  on  how  the  set  of  skills  varied  between  each 
level  of  leadership  rather  than  how  specific  skills  varied,  an  exact  one-to-one  comparison 
was  not  accomplished.  However,  the  emphasis  on  which  leadership  skills  were  observed 
to  be  most  useful  for  entry  and  middle  level  managers  was  compared  with  the  results  of 
this  study. 

For  entry  level  managers,  this  study  highlights  the  importance  of  personality, 
effort,  and  attitude  compared  with  the  strataplex  model’s  larger  emphasis  on  cognitive 
skills  followed  by  interpersonal,  business,  and  strategic  skills,  respectively.  More 
specifically,  it  appears  this  study’s  emphasis  on  the  contributions  of  high  effort,  role 
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modeling  (honesty,  integrity,  idealized  influence),  and  attitudinal  effects  are  obscured  in 
the  strataplex’s  model  for  entry  level  leadership.  Moreover,  the  strataplex’s  heavy 
emphasis  on  collecting,  processing,  and  disseminating  information  (cognitive)  appears  to 
have  little  prominence  in  the  current  study’s  findings.  Table  14  highlights  the 
differences  between  each  study  for  entry  level  leadership. 


Entry  Level  Leadership 

Strataplex 

Current  Study 

Cognitive  Skills 

Personality 

Emphasis;  High 

Conscientousness 

Verbal  Communication 

Honesty 

Written  Communication 

Idealized  Influence 

Reading  Comprehension 

Integrity 

Active  Listening 

Mastery  Orientation 

Active  Learning 

Verbal  Communication 

Critical  Thinking 

Interpersonal  Skills 

Effort 

Emphasis:  Medium  Low 

Engagement 

Social  Judgment 

Effort 

Social  Complexity 

Human  Relation  Skills 

Attitude 

Social  Perceptiveness 

Counterproductive 

Coordination 

Work  Behavior 

Negotiation 

Negative  Affect 

Persuasion 

Business  Skills 

Emphasis;  Low 

Management  of  resources 

Strategic  Skills 

Emphasis:  Low 

Visioning 

Systems  Perception 

Evaluate  alternatives 

Table  14:  Strataplex  Comparison  of  Entry  Level  Leadership  Criteria 


Regarding  middle  level  managers,  the  strataplex  model’s  emphasis  on  cognitive 
skills  seems  to  correspond  with  this  study’s  finding  of  developing  team  members  and 
verbal  communication  as  it  relates  to  collecting,  processing,  and  communicating  mental 
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models  to  others.  Additionally,  while  this  study’s  finding  that  ability  to  aceommodate 
others  is  not  explieitly  observed  in  the  strataplex  model,  it  appears  a  similar  consideration 
for  others  is  utilized  in  the  strataplex ’s  composite  of  interpersonal  skills.  However,  this 
study’s  emphasis  on  cognitive  adaptability,  or  adaption  to  external  factors  does  not 
appear  to  be  as  accentuated  in  the  strataplex’s  model  for  middle  level  managers.  Table 
15  shows  the  differences  between  each  study  for  middle  level  managers. 


Middle  Level  Leadership 

Strataplex 

Current  Study 

Cognitive  Skills 

Trust 

Emphasis:  High 

Trustworthiness 

Verbal  Communication 

Verbal  Communication 

Written  Communication 

Selflessness 

Reading  Comprehension 

Developing  Team  Members 

Active  Listening 

Active  Learning 

Accommodation 

Critical  Thinking 

Followership 

Interpersonal  Skills 

Introspection 

Emphasis:  Medium  Low 

Feedback  Acceptance 

Social  Judgment 

Social  Complexity 

Adaptability 

Human  Relation  Skills 

Social  Perceptiveness 
Coordination 

Negotiation 

Persuasion 

Cognitive  Adaptability 

Business  Skills 

Emphasis:  Low 

Management  of  resources 

Strategic  Skills 

Emphasis:  Low 

Visioning 

Systems  Perception 

Evaluate  alternatives 

Table  15:  Strataplex  Comparison  of  Middle  Level  Leadership  Criteria 


While  the  strataplex  model  contributes  to  the  understanding  of  how  different  skill 
sets  change  throughout  the  levels  of  organizational  leadership,  it  does  not  address  how 
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specific  skills  change.  This  study  empirically  identified  the  most  important  dimensions 
for  both  entry  and  middle  level  managers  under  maximum  performance  conditions. 

Since  the  strataplex  study  did  not  highlight  skills  that  specifically  related  to  maximum 
performance,  the  risk  of  losing  a  degree  of  fidelity  in  the  comparison  is  noted.  However, 
the  comparison  made  was  only  intended  to  determine  if  there  were  any  significant 
differences  between  the  two  studies.  As  such,  it  appears  this  study  uncovered  additional 
dimensions  that  suggest  an  alternative  structure  of  leadership  effectiveness  for  entry  and 
middle  level  managers.  Nevertheless,  this  study’s  alternate  findings  similarly  contribute 
to  the  understanding  of  leadership  effectiveness  at  different  levels  of  the  organization. 

Implications 

This  study  benefits  both  researchers  and  practitioners  alike.  For  researchers,  a 
clearer  understanding  of  leadership  criteria  for  entry  level  managers  (i.e.  leadership  of 
self)  and  middle  level  managers  (i.e.  leadership  of  teams)  exposed  to  "can  do"  maximum 
performance  conditions  is  achieved.  However,  this  research  also  benefits  practitioners 
who  gain  insight  on  matters  of  selection,  promotion,  and  placement  of  such  managers  in 
their  organization. 

As  a  representation  of  entry  level  management  leadership  criteria  under  maximum 
performance  conditions,  this  study  suggests  that  managers  who  are  motivated  to  gain 
competency,  abide  by  moral  principles,  try  hard,  and  have  a  positive  attitude  are 
demonstrating  the  most  important  antecedents  of  their  leadership  effectiveness. 
Additionally,  the  majority  of  the  most  important  dimensions  are  within  the  domain  of  an 
individual  and  simultaneously  capable  of  standing  alone  outside  the  effects  of  external 
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stimuli  (sans  idealized  influenee  and  verbal  eommunieation).  Thus,  the  eriteria  of 
personality,  effort,  and  attitude — or  leadership  of  self,  serve  as  an  effeetive  framework  to 
lead  as  an  entry  level  manager. 

Regarding  middle  managers,  one  who  is  trustworthy  and  selfless,  effeetive  in 
developing  team  members,  able  to  follow  others  as  well  as  lead,  aeeepting  of  feedback, 
and  able  to  adapt  to  environmental  changes  will  be  exhibiting  antecedents  considered 
most  important  to  this  level  of  management.  Thus,  it  seems  the  most  important  middle 
management  dimensions  that  define  the  leadership  effectiveness  criteria  of  trust, 
accommodation,  and  adaptability  heavily  pertain  to  the  individual’s  interaction  with 
others  and  the  environment.  That  is,  the  “leadership  of  teams”  framework  seems  to 
appropriately  capture  the  leadership  effectiveness  criteria  for  middle  level  managers 
exposed  to  maximum  performance  conditions.  Moreover,  considering  that  leadership  is 
the  ability  to  influence  others  to  achieve  organizational  goals,  it  appears  the  interaction 
between  the  individual  and  others  is  representative  of  leadership  given  the  leadership 
context  in  which  the  dimensions  were  formulated.  That  is,  leadership  dominates  nearly 
all  of  the  most  important  dimensions  for  middle  level  managers.  In  summary,  leadership 
is  a  dimension  of  performance  for  entry  level  managers  but  is  performance  for  middle 
level  managers. 

Limitations 

Military  samples  have  frequently  been  used  in  leadership  research  to  include  the 
development  of  common  leadership  assessment  tools  (Northouse,  2001).  However,  when 
using  a  military  sample,  the  possibility  that  external  validity  can  potentially  be 
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compromised  exists  and  thus,  applieability  to  private  sector  may  be  jeopardized.  This 
consideration  was  hoped  to  be  mitigated  given  the  broad  range  of  job  specialties  and 
backgrounds  of  ASBC  and  SOS  students.  Nevertheless,  a  more  diverse  sample  that 
includes  appropriate  representation  of  private  sector  employees  might  be  warranted. 

Although  care  was  taken  to  ensure  quality  dimension  development — a 
fundamental  concern  in  whieh  assessment  eenters  ean  fall  short  (Lievens  &  Conway, 
2001) — there  are  potential  weaknesses  in  the  method  chosen  to  achieve  this.  Bias  is  a 
potential  problem  that  tends  to  surface  when  using  surveys  to  eollect  data  (i.e.  self- 
report).  Collecting  data  through  self-reports  is  common,  practical,  and  sometimes  the 
only  appropriate  means  of  data  collection,  especially  in  the  organizational  behavior  field. 
Its  use  has  been  extensive  (Sackett  &  Larson,  1990;  Donaldson  &  Grant- Vallone,  2002) 
and  is  more  than  likely  to  remain  a  preferred  method  to  gather  data  despite  its  eriticism  as 
a  potential  research  weak  point  due  to  problems  with  common  method  bias  (Campbell, 
1982;  Podsakoff,  Organ,  1986).  Although  this  study  eolleeted  self-report  data,  this 
eoncem  was  believed  to  be  mitigated  by  using  a  mixed  methods  approach  and 
implementing  a  variety  of  remedies  proposed  by  Podsakoff,  MaeKenzie,  Lee,  and 
Podsakoff  (2003).  For  example,  using  both  surveys  and  focus  groups  as  a  means  of  data 
collection  reduced  the  potential  of  a  eontaminated  proeess  that  a  single  method  approach 
may  be  more  susceptible.  Accordingly,  obtaining  data  using  a  mixed  methods  approaeh 
improves  the  substantiality  of  the  data  by  eonverging  findings  from  two  methods  instead 
of  one  and  thus  further  mitigates  the  potential  effects  induced  by  common  method  bias 
(Creswell,  2002).  Moreover,  common  method  bias  remedies  were  implemented  to 
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include  collecting  data  from  numerous  experts,  using  items  derived  from  instructor  inputs 
as  a  means  of  increasing  both  accuracy  and  familiarity,  counterbalancing  question  order, 
implementing  a  time  delay  between  sending  surveys,  and  retaining  the  anonymity  of 
those  filling  out  the  surveys  were  among  the  remedies  implemented  (Podsakoff  et  ah, 
2003).  Employing  these  remedies  was  considered  to  be  an  effective  countermeasure  to 
mitigate  the  top  biases  associated  with  collecting  data  from  experts  such  as  anchoring, 
overconfidence,  and  availability  (Keeney,  et  al,  1991).  Additionally,  given  the  expert 
population  receiving  the  survey,  it  was  hoped  that  any  interpretation  issue  would  be 
minimal  to  the  content-familiar  instructor  sample. 

Regarding  the  focus  groups,  there  is  additional  chance  that  expert  biases  may  be  a 
factor  when  contributing  inputs  in  a  public  forum.  However,  the  nominal  group 
technique  was  used  to  not  only  attain  expert  consensus,  but  reduce  the  chance  for  such 
biases  to  occur  by  enabling  all  contributors  an  equal  chance  to  voice  their  inputs  both 
individually  and  collectively  as  a  group  and  thus,  elicit  a  more  comprehensive 
perspective  of  leadership  effectiveness  while  avoiding  groupthink  (Heldman,  2005). 
Nevertheless,  there  is  a  potential  that  social  desirability  or  undesirability  of  proposed 
performance  inputs  may  have  affected  the  contributions  that  were  received.  Another 
possible  drawback  to  the  focus  group  approach  used  was  the  inability  to  randomly  select 
instructors  to  contribute  to  focus  group  discussions.  Instructors,  although  qualified  to 
contribute,  were  subordinate  to  their  schedule  availability  which  may  have  influenced  the 
discussions. 
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Another  potential  limitation  arising  out  of  the  method  used  to  derive  the 
leadership  criteria  is  the  issue  of  content  validity.  Although  the  formulation  of 
dimensions  using  expert  inputs  was  considered  to  be  a  strong  indicator  of  appropriately 
developed  leadership  criteria,  it  is  quite  possible  that  the  variety  of  data  collected  is  an 
incomplete  set.  Therefore,  it  is  possible  the  results  of  the  exploratory  factor  analysis  may 
have  yielded  inconclusive  results.  Additionally,  the  dimensions  identified  have  not  been 
exposed  to  reliability  or  validity  tests.  Moreover,  it  was  assumed  that  the  assessment 
center  exercises  were  appropriately  designed  to  elicit  leadership  effectiveness  but  perhaps 
this  assumption  should  be  verified  through  an  exercise  analysis. 

Future  Research 

Regarding  generalizability,  it  would  be  useful  to  compare  these  results  with  a 
study  of  private  sector  assessment  centers  that  similarly  identified  leadership  criteria 
under  maximum  performance  conditions.  Not  only  would  such  a  study  shed  light  on  the 
external  validity  of  these  results,  but  it  would  provide  more  clarity  to  the  applicability  of 
leadership  criteria  at  the  less  studied  entry  and  middle  management  levels.  Additionally, 
increasing  the  sample  size  and  varying  the  source  of  the  collected  data  by  considering  not 
only  experts,  but  trainees  (i.e.  novices)  would  be  an  interesting  exploration  of  the 
perceptual  differences  of  leadership  criteria  between  these  sources.  However,  it  would 
also  be  interesting  to  incorporate  the  perspective  of  senior  level  managers  and  compare 
their  view  of  entry  and  middle  level  management  leadership  effectiveness  with  the 
findings  of  this  study. 
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As  Taylor  and  Small  (2002)  point  out,  one  of  the  proposed  implieations  of 
identifying  maximum  performance  criteria  is  the  difficulty  of  applying  a  similar  degree  of 
motivated  behavior  to  typical  performance,  or  day-to-day  work  environment.  Thus, 
identifying  leadership  measures  under  typical  performance  conditions  for  entry  or  middle 
managers  may  help  shed  light  on  the  ability  to  relate  maximum  performance  leadership 
measures  to  typical  performance  leadership  measures. 

Furthermore,  it  would  be  beneficial  to  confirm  the  validity  and  reliability  of  the 
identified  leadership  criteria.  Additionally,  investigating  the  predictive  ability  of  the 
identified  leadership  dimensions  would  be  of  great  interest  to  both  academics  and 
practitioners  alike.  Thus,  a  study  that  establishes  criterion  validity  would  be  of  great  use. 
Finally,  undergoing  such  a  process  would  alleviate  validity  concerns  that  surround  much 
of  the  assessment  center  literature. 
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